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Abstract —The coordination of autonomous agents is a critical 
issue for decentralized communication networks. Instead of trans¬ 
mitting information, the agents interact in a coordinated manner 
in order to optimize a general objective function. A target joint 
probability distribution is achievable if there exists a code such 
that the sequences of symbols are jointly typical. The empirical 
coordination is strongly related to the joint source-channel coding 
with two-sided state information and correlated source and state. 
This problem is also connected to state communication and is 
open for non-causal encoder and decoder. We characterize the 
optimal solutions for perfect channel, for lossless decoding, for 
independent source and channel, for causal encoding and for 
causal decoding. 

Index Terms —Shannon Theory, State-dependent Channel, 
Joint Source-Channel Coding, Empirical Coordination, Empiri¬ 
cal Distribution of Symbols, Non-Causal Encoding and Decoding, 
Causal Encoding, Causal Decoding. 

I. Introduction 

The problem of the coordination of autonomous agents 
is the cornerstone of decentralized communication networks. 
Communication devices are considered as agents that interact 
in a coordinated manner in order to achieve a common 
objective, for example, the transmission of information. This 
analysis is based on a two step approach Q, 0. The first 
step is the characterization of the set of achievable joint prob¬ 
ability distributions over the symbols of source and channel. 
The second step is the maximization or the minimization of 
an objective function (source distortion or channel cost) by 
considering the set of achievable joint probability distributions. 

Empirical coordination has been investigated in El, a with 
a rate-distortion perspective. In the literature of game theory, 
the agents coordinate their actions by implementing a coding 
scheme that satisfies an information constraint 0. Empirical 
coordination was under investigation in a for perfect channel, 
in 0 for causal encoding, in 0 with channel feedback, in 
0 for a multi-user network with an eavesdropper, in EO) 
for polar codes, in 0 for causal decoding, in ifTTI . ifT^ with 
feedback from the source, and in for lossless decoding 
and correlated source and state. 

The characterization of the set of achievable joint proba¬ 
bility distributions is equivalent to the joint source-channel 
coding with two-sided state information El, m and corre¬ 


lated source and state (Eig. [U. It is also related to the problem 
of state communication ifThl . In this paper, we investigate the 
empirical coordination with non-causal encoding and decoding 
and we characterize the optimal solutions for three particular 
cases: perfect channel, lossless decoding and independent 
source and channel. We also characterize the optimal solutions 
for causal encoding and causal decoding with two-sided state 
information. 





Fig. 1. Non-causal encoding / : X S" —> X" and decoding g : 

yn ^ yn functions for i.i.d. source "Pusz and channel 7j)|xs- 

Channel model and definitions are presented in Sec. [Ill In 
Sec.|IIIl we provide achievability and converse results for non- 
causal encoding and decoding. In Sec. IIVI we characterize 
optimal solutions for perfect channel, for lossless decoding 
and for independent source and channel. Causal encoding and 
decoding are presented in Sec. |V] Conclusion is in Sec. |Vl] 
and sketches of proof are provided in App. lAl iBl lO iDl lEl 

II. System Model 

The problem under investigation is depicted in Eig. E] Cap¬ 
ital letter U denotes the random variable, lowercase letter u G 
U denotes the realization and (Y" denotes the n-time cartesian 
product. ?7", S'", X", X”, Y", Y" denote the sequences of 
random variables of source symbols u" = (ui,..., u„) G (Y", 
of channel states G S^, of state informations at the decoder 
zTi g 2:"^ of channel inputs a;" G T", of channel outputs 
y" G y" and of outputs of the decoder u" G V". Sets 
YY, S, Z, X, y, V are discrete. A(A’) stands for the set of 
probability distributions 'P{X) over X. The total variation 
distance between the probability distributions Q and V is 
denoted by ||Q-P||tv = l/2'X]a;eY \Q{^) Notation 

t{y\x) denotes the indicator function, that is equal to 1 if 
y = X and 0 otherwise. Markov chain property is denoted 
by Y -e- X -e- U and holds if for all {u, x, y) we have 
P(y|x,M) = V{y\x). Information source and channel states 











are correlated and i.i.d. distributed with Vusz- Channel is 
memoryless with transition probability 7^|xs- Statistics of Vusz 
and Ty\y,s ^6 known by both encoder C and decoder V. 


Definition II.l Non-causal code c = (f, g) G C (n) is defined 

^ A-", (1) 

5 : 3^" X Z” —^ V". (2) 

A/(rt|zi") denotes the occurrence number of symbol u G U in 
the sequence rt". The empirical distribution Q" G A(U x S x 
Z X X xy xV) of (m”, s", z", x", y”, t;") is defined by: 


Q’"{u,s,z,x,y,v) 

y{u,s,z,x,y,v) 


N{u, s, z, X, y, v\u^, s”,z^,x”, y", w") 


G U X S X Z X X xy xV. 


(3) 


Fix a targetproba. distribution Q G A{UxSxZxXxyxV), 
the error probability of the code c G C(n) is defined by: 


Ve{c) = Vc 




(4) 


3) The joint probability distribution 7^u5z(m, s, z)(8>Q(a::|u, s)® 
T{y\x, s) Z> Q(v\u, s, z, X, y) is not achievable if: 

in&x(^I{Wi]Y,Z\W2)-I{W2]U,S\Wi^ <0, (8) 

Q is the set of distributions Q G A{fA x S x Z x Wi x W 2 x 
X xy xV) with auxiliary random variables (Vki, W 2 ) s.t.: 

'E(^i.^ 2 )ewi X W 2 wi,W 2 ,x, y, v) 

= Vusziu, s, z) 0 Q{x\u, s) ®T{y\x, s) 0 Q{v\u, s, z,x,y), 

< y ^ (X, 5) ^ {U, Z, ICi, W 2 ), 

Z ^ {U, S) ^ {X, Y, Wi, W 2 ), 

V ^iY,Z,WuW2) ^ {U,S,X). 

The probability distribution Q G Q decomposes as follows: 
Pusz{u,S,z) 0 Q{x,W\,W 2 \u, s) ®T{y\x,s) 0 Q{v\y,Z,W\,W 2 )- 

The supports of the auxiliary random variables (lki,W 2 ) 
are bounded by max(|>Vi|, |>V 2 |) < (|5| + 1) ■ (|S| + 2) with 
B = U xS X Z X X xy xV. 


where Q" G AiUxS X Z X X X y X y ^ IS the random variable 
of the empirical distribution induced by the code c G C{n) and 
the probability distributions Vusz, E|xs- 


Definition II.2 Probability distribution Q G A{U x S x Z x 
X xy xV) is achievable if for all e > 0, there exists a n G N 
s.t. for all n>n, there exists a code c G C(n) that satisfies: 


Ve{c) = Vc 




< e. 


(5) 


If the error probability Ve(c) is small, the empirical frequency 
of symbols {u,s,z,x,y,v) G lA x S x Z x X x y xV m close 
to the target probability distribution Q{u, s, z, x, y, v), i.e. the 
sequences (C/", S'", Z", X”, F", 1/”) G A*"(Q) are jointly 
typical, with large probability. In that case, the sequences of 
symbols are coordinated empirically. 

The performance of the coordination is evaluated by an 
objective function $:Z^xSxZxA’xy’xVi->'R, as 
stated in El and Uni. This approach is powerful and since the 
expectation E[(i(M, u)] over the set of achievable distributions, 
provides directly the minimal distortion level d{u, v). This is 
valid for any function, as the channel cost c{x) or the utility 
$(rt, s, z, X, y, v) of a decentralized network El. 


III. Achievability and Converse Results 
We provide necessary and sufficient conditions for non- 
causal encoding and decoding. This problem is open. 


Theorem III.l (Non-Causal Encoding and Decoding) 

1) If the joint probability distribution Q{u, s, z,x,y,v) is 
achievable, then it satisfies the marginal and Markov chains: 


Q{u, s, z) = Vusziu, s, z), Q{y\x, s) = T{y\x, s), 
Y^{X,S)^{U,Z), Z ^ {U,S) ^ {X,Y). 


Remark III.2 It is possible to send an additional message 
m G A4 with rate R corresponding to left-hand side of equa¬ 
tion while still satisfying the coordination requirement. 
This remark also extends to the other results of this article. 

Sketches of proof are available in App. and |B] Achiev¬ 
ability result of Theorem llll.ll is based on hybrid coding HU, 
ESI and is stated in m, with a unique auxiliary random 
variable W = {Wi,W 2 ), without state informations S and 
Z. Empirical coordination is obtained by considering (C7, S) 
as state information at the encoder and (Y, Z) as a state 
information at the decoder. This open problem is also related 
to "state communication" EH which is solved for Gaussian 
channels EH, but remains open for non-causal encoding and 
decoding. In the following, we connect the duality result of 
ESI and the separation result of El to empirical coordination. 

IV. Optimal Solutions for Particular Cases 

In this section, we characterize the joint probability distri¬ 
butions that are achievable for three particular cases. 


A. Perfect Channel is defined by Ty\y.s = l(y|T) as in Fig. 
Z" 



Theorem IV.l (Perfect Channel) 

1) If the joint probability distribution Q(u, s, z,x,v) is 
achievable, then it satisfies: 

Q{u,s,z) =Vusz{u,s,z), Z-e-{U,S)-e-X. (9) 


2) The joint probability distribution Vusziu, s, z)Z)Q{x\u, s)0 
T{y\x, s) ® Q{v\u, s, z, x, y) is achievable if: 


2) The probability distribution Vusz{u, s, z) 0 Q{x\u, s) ® 
Q{v\u,s,z,x) is achievable if (1 10b . The converse holds. 


max 

QgQ 


I{WuW 2-,Y, Z) - I{Wi,W2-, U, S) 


max 

Q^Qp 


I{W2\Z\X) + H{X)-I{X,W2\U,S)] >0, (10) 


> 0 , 


(7) 






















Qp is the set of distributions Quszw 2 xv with W 2 satisfying marg¬ 
inals and Z-e-{U, S)-e-{X, W 2 ) and V-e-{X, Z, W 2 )-e-{U, S). 
Support satisfies \W 2 \ and Q G Qp decomposes: 

Pli5z(m, S, z) ® Q{x\u, s) ® Q{w2\u, S, x) ® Q{v\x, Z, W2)- 


Achievability comes from replacing Y and Wi by X in 
Theorem IIII. 1 1 and converse is in App. ICl This results extends 
tbe coding theorem of Wyner-Ziv ll20l with distortion d(u, v), 
to the framework of coordination. The message of rate R 
corresponds to the channel inputs X" of rate log 2 \X\ and 
the optimal distortion level D* is obtained by taking the ex¬ 
pectation u)], as in Corollary IV.2I The main difference 

is that the symbols (C7, S, V) are coordinated with X. 

B. Lossless Decoding defined by Q{v\u, s, z,x,y) = l('u|u) 
The characterization for lossless decoding with correlated 
source and state is in ini. Achievability is also a particular 
case of Theorem mu] by replacing random variables V 
and W 2 by U. Joint distribution Vu 5 z{u, s, z) ® Q{x\u, s) ® 
T{y\x, s) ® l('u|ii) is achievable if (HB- The converse holds. 


max{l{U,Wi;Y,Z)-I{Wi-,S\U)-H{U)] >0, (11) 

SeQi \ / 

Qi is the set of distributions Quszwixyu satisfying marginal 
conditions and Markov chains Y -e- {X,S) -e- (U,Z,Wi) 
and Z -e- (17,5') -e- (X,Y,Wi). This result extends the 
coding theorem of Gel’fand-Pinker ||2TI to the framework of 
coordination. The message is a sequence of source symbols U 
correlated with the states (5, Z) and channel inputs X. Duality 
between equations (fTOl i and (fTTTi recalls the duality between 
channel capacity and rate distortion, as mentioned in 03. 


C. Independent source {U,Z^V), channel {S,X,Y), Tig. [J] 

Zn 



Fig. 3. The random variables of the source (U, Z, V) are independent of 
the random vaiiables of the channel (S, X, y). 


Theorem IV.2 (Independent Source and Channel) 

1) If the product Q(u,z,v) ® Q{s,x,y) of probability 
distribution is achievable, then it satisfies: 

Q{u,z) = Tuz{u,z), Q(s)='Ps(s), Q{y\x,s) = T{y\x,s). (12) 

2) Probability distribution Vuziu, z) ® Q{v\u, z) 0 'Ps(s) 0 
Q(x|s) 0T(t/|at, s) is achievable if (I131 l. The converse holds. 

m&x(l{Wi-,Y) + I{W2-,Z)-I{Wi-,S)-I{W2-,U)) >0, (13) 

Q £ Qs y y 

Qs is the set of product distributions Quzw 2 v 0 Qsxwiy with 
auxiliary random variables (fki, W 2 ) satisfying conditions: 

Z -c^U ^W2, V ^{Z,W2)^U, y^(X,5)-e-VFi. 

Supports of {Wi, W 2 ) are bounded by (|S| + 1) • (|i3| -f 2). 
The product distribution Q € Qs decomposes as follows: 

Puz{u, z) 0 Q{w 2 \u) ® Q{v\z, 1 x 2 ) <S) Ps{s) <Si Q{x, WI |s) ® T{y\x, s). 


As mentioned in m, the independence between the prob¬ 
ability distributions of the source and channel induces the 
separation of the source coding and the channel coding. This 
result is related to Theorem 1 in na, with expected distor¬ 
tion d{u, v) and channel cost c{x) functions. By considering 
{U, Z, W 2 , V) independent of (5, X, Wi, Y) and introducing 
three indexes (m, l,j) with rates (Rm, R;, Rj) satisfying, 

Rm + Ri> I{W2-,U), Ri <I{W2;Z), 

Ry > 7(lTi; 5), R™ -b Ry < I{Wv, Y), 

the achievability of Theorem llll. 1 [ becomes a separated coding 
scheme. The author would like to thank Pablo Piantanida, 
Matthieu Bloch and Claudio Weidmann for useful discussions 
about the independence of the auxiliary random variables 
(VPi, W 2 ) in the converse proof of Theorem II V. 21 

V. Causal Encoding and Decoding 
The encoding (resp. decoding) is causal if for all i G 
{1,... ,n}, we have Xy = fi{U\S^), (resp. Vi = gi{Y\ Z^)). 
We characterize of the set of achievable joint probability 
distributions for causal encoding and for causal decoding. 

Z-n. 



Fig. 4. Causal encoding function /y : W® X <S® —>■ X, for alH g {1,..., n}. 


Theorem V.l (Causal Encoding and Non-Causal Decoding) 

1) If Q{u, s, z, X, y, v) is achievable, then it satisfies: 

I Q(u,s,z) = Pusz(u,s,z), Q(y\x,s) = T(y\x,s), 

\ Y^(X,S)^(U,Z), Z^(U,S)^(X,Y). 

2) The distribution Vusz{u, s, z) 0 Q{x\u,s) 0 T{y\x,s) 0 
Q{v\u, s, z,x,y) is achievable if ( fTSl ). The converse holds. 

I{WuW2;Y,Z)-I{W2;U,S\Wi)^ >0, (15) 

Qe is the set of joint distributions QuszwiW 2 xyv with auxiliary 
random variables (WijlJQ) satisfying marginal cond. and: 

f (C, 5) independent of Wi, X ^ (U, S,Wi) ^ W 2 , 

I y ^ (X, S') ^ (U, Z, Wi, W 2 ), 

' Z^{U,S)^ (X, y, Wi,W2), 

V^iY, Z, Wi, W 2 ) ^ (U, S, X). 

Supports of {Wi, W 2 ) are bounded by {\B\ 1) ■ (|S| -I- 2). 

The probability distribution Q G Qe decomposes as follows: 

Tusz 0 Qwi 0 Qw2juswi Z) Qx|u5Wi Z ^|xs ® Qv|yzwiW2- 

Proofs are available in App. s and |B] This result is a 
particular case of Theorem 2 in Q, by considering {U, S) as 
information source and (Y, Z) as channel output. Empirical 
coordination is equivalent to joint source channel coding with 
two-sided state information and correlated source and state. 
Theorem I V. 1 1 also reduces to Theorem 3 in na, by considering 
([/, 5) as channel state and {Y, Z) as channel output. 


max 

QeQe 




























When considering strictly causal encoding instead of causal 
encoding, the optimal solution is obtained by replacing Wi by 
X in equation (fTSt . 

Corollary V.2 (Causal Encoding without Coordination) 

Consider a distortion function d : if x V i—> M. The distortion 
level D > 0 is achievable if and only if: 

max {i{Wi,W2\Y,Z)-I{W2\U,S\Wi)] > 0 , 

Qwy ,Qw2|uSWy ’^x|uSWy ’ \ / 

Qv!yzwiw2 .E[dC(7,V)]<D 

Corollary IV. 2 1 is a direct consequence of Theorem IV. II The 
distortion level D > 0 is achievable if and only if there exists 
a joint distribution Vusz ® Qwi ® Qw 2 |uswi ® Qx|u 5 wi ® T^lxs ® 
Qv|yzwiw 2 that satisfies (fT^ . such that E[d((7, V)] < D. Similar 
analysis is mentioned in ||2l for the proof of Theorem V.3. 

Causal decoding and non-causal encoding is considered in 
El without S and Z. Joint probability distribution Vusz ® 
2x|us ® %\zs ® Qv|uszxy is achievable if (fT^ . Converse holds. 

max (l{Wi]Y,Z\W 2 )-I{Wi,W 2 -,U,S)]>Q, (16) 

^vjyzw2 

Remark that V depends on (V, Z, W 2 ) but not on W^. When 
considering strictly causal decoding instead of causal decod¬ 
ing, the optimal solution is obtained by replacing W 2 by V in 
equation (flbl l. More details are provided in El- 

VI. Conclusion 

The problem of empirical coordination is closely related 
to the joint source-channel coding with two-sided state in¬ 
formation with correlation between source and state. These 
two problems are also related to state communication. We 
provide achievability and converse results for the non-causal 
case, that is open. We characterize the optimal solutions for 
perfect channel, for lossless decoding, for independent source 
and channel, for causal encoding and for causal decoding. 


jointly typical. Decoder returns V” drawn from 
depending on (V", Z", Wf{m), W^(m)). 

The pair (lVi,W 2 ) can be replaced by a single W. From 
properties of typical sequences, packing and covering Lemmas 
stated in ll23l pp. 27, 46 and 208, equations (fTTl i. (flST l imply 
that the expected probability of error is bounded for all n > n: 

S"-) g 2l*"(Q))j < e, (19) 

Ec^-p(vm€M, (U^,S^,W^(m),W^(m)) ( < e, ( 20 ) 

Ec^-p(3m' ^ m, s.l. (V" , Z" , WP (m'), Wj-(m')) £ A*" (Q)^ j < e. (21) 

There exists a code c* € C(n) such that sequences are jointly 
typical for distribution 7^jsz<8i Qxwiw2|js®7^|xs® Qv|yzwiw 2 with 
probability more than 1 — 3^. 

B. Sketch of Converse of Theorem I///. 7 1 

Consider code c(n) C C with small error probability Veic). 

i = l 

- ( 22 ) 

i = l 

= (23) 

1=1 1=1 

< n ■ maxi I(Wi-Y,Z\W 2 ) - I{W 2 -,U,S\Wi)]. (24) 

eeQ V / 

Eq. (I 22 I 1 is due to Csiszar Sum Identity and properties of MI. 
Eq. (|2^ introduces auxiliary random variables Wi i = 
(c/f+i, Zf+i) and satisfying: 

r, ^ (W„ S,) ^ {Uu Z,, Wi, 2 , 1 ^ 2 , 2 ), (25) 

Z, ^ {U,,Si) ^ {X„ Tj, Wi,2, 1 ^ 2 , 2 ), (26) 

V 2 ^ {¥„ Z„Wl,„W2,^) ^ iU,,S,,X,). (27) 

This is due to memoryless channel, i.i.d. source and non-causal 
decoding since is included in (Vi, Zi,Wi^i,W 2 ^i). 

Eq. (I 24 I 1 comes from taking the maximum over the set Q. 

C. Sketch of Converse of Theorem \IV.1\ 

Consider code c(n) C C with small error probability Veic). 


Appendix 

The full versions of the proofs are stated in EH- 

A. Sketch of Achievability of Theorem 17/7. 7 1 

Consider Q{u,s,z,wi,W 2 ,x,y,v) G Q that achieves the 
maximum in ©. There exists 5 > 0 and rate R > 0 such that: 

R > I{WuW2-,U,S)+5, (17) 

R < I{WuW2-,Y,Z)-S. (18) 

• Random codebook. We generate |AJ| = 2”*^ pairs of 

sequences W^(to)) with index m € A4 drawn 

from the i.i.d. marginal probability distribution 

• Encoding function. Encoder finds the index m € AA such 

that {U^,S^,Wf{m),W^im)) € A*"(Q) are jointly 
typical. It sends X" drawn from depending on 

• Decoding function. Decoder finds the index m € Ai 
such that {Y^,Z^,Wf{m),W:^{m)) G A*^{Q) are 


= 

H{X^, Z 



H{X^, Z’ 


‘,S”) (28) 

< 

f:mxi 

i=l 

i = l 

\Z^-,Ui,Si\U’fi,S’i 

Vi) 


- 


”1(7”,S”) 




(29) 

= 

f^zcx- 

i = l 

,Z".C/”+i.Sr+i 

;Xi,Zi) 




- 

i=l 

-z'.up+i.sr+i 

;Ui,Si) - 

H(X'',Z' 


' ,S") (30) 

< 

f:Hx- 

i = l 

,z-\uAi,sT+ 

iWi.Zi) 




- 

f:iix- 

i = l 

,z-\uAi,sT+ 

i;C/i,Si) 



(31) 

= 

flnxi. 

i = l 

W2,i-,Xi,Zi) - 

I(Xi,W2, 

i;Ui,Si) 


(32) 

< 

n • max | 
QGQp ' 

^/(X. tV2;X, Z) 

- I(X, W 2 ; U, S)J . 


(33) 


Eq. (l28l l. (l29l l are due to properties of mutual information. 
Eq. (EOjl is due to the i.i.d. source {U,S). 

Eq. (ISTT i is due to the i.i.d. source ((7, S, Z): 77(Z”|(7", 5'") = 








E H{z,\u,, s,) = E H{Z,\X^, z-\ C/El, E"+1, c/., s,). 

Eq. (l3^ introduces auxiliary random variable W 2 ,i = 
(X“*, C/jEi, E’+i) *^hat satisfies Markov chains; Zi -e- 
(C/„ 5,) ^ {X,,W 2 .^) and E ^ (X„ W 2 .,^) ^ (C/„ S,). 

This is due to i.i.d. source (C/, 5, Z) and non-causal decoding. 
Eq. (l33l l comes from taking the maximum over the set Qp. 

D. Sketch of Achievabilitv of Theorem \V.l\ 


Eq. (36 is due to Csiszar Sum Identity and properties of MI. 
Eq. (I371 i introduces auxiliary random variables Wi t = 
(C/*-\ E-i) and W 2 ,^ = (E+i, E”+i) satisfying: 


(Ui, Si) are independent of Wi^i, (39) 

Xi -e- {Ui,Si,Wi,i) -e-W2,i, (40) 

Yi -e- iXi,Si) -e- (Ui, Zi,Wi,i,W2,i), (41) 

Zi -e- iUi,Si) -e- {Xi,Yi,Wi,i,W 2 ,i), (42) 

Vi -e- (Yi, Zi,Wi,i,W2,i) -e- (Ui,Si,Xi). (43) 


Consider Q G Qe that maximizes O and block-Markov 
code c G C(n) over B € N blocs of length n G N with; 

R > I(W2;U,SlWi)+S, (34) 

R < I(Wi;V,Z) + I(W2;V,ZlWi)-S. (35) 

• Random codebook. We generate \Ai\ = 2"*^ sequences 
W^{m) drawn from Q®" with index m € Ai. Eor each 
m G A4, we generate the same number \M\ = 2”*^ of 
sequences W^im^rh) with index fh G Ai, drawn from 

depending on W{^(m). 

• Encoding function. It recalls rrib-i and finds mf, G A4 s.t. 

sequences Sf_^,Wf{rnb-i),W^{mb-i,mb)) G 

^£"(2) ^te jointly typical in block b — 1. It deduces 
Wf(^mb) for block b and sends XJ(^ drawn from 
depending on (C/^, Sf, Wf{mb)). 

• Decoding function. It recalls nib-i and finds mb G Ai 

such that (Yf^_^,Z^_^,Wf{mb-i),W:f{mb-i,mb)) G 
yl*"(2) and (E",Z”,IEi”(m6)) G ^^"(2) are jointly 
typical. It returns E-i drawn from 2®"^^,,^^,,^ depending 
on {Yff^,Z^_„Wf{mb-i), {mb-V,ZT)) ■ 


/rrn Qn\ 1^ 2|6 1| h |6-|-1| | 

IE" 1—■ I. ■ !■ ■. ■ !■ ■ ■. 


IE" 1 - 1 —43^: - 1 1 


1 1 1 " 1 1 


fyn 1 1 - ^ . k . .- 1 


ij. ^ 1 iiiiiiiiiiiiiiii 

- 1 1 1 V 1 1 


El 1 1 " ^ ^ 


. . T. . ' 


Equations (l3^. (l35]l imply for all n > h and for a 1 
number of blocks B G N, the sequences are jointly typ 
with large probability: 

Ec['p((U", S") i < e, 

irge 

ical 


Ec['p(vm e At.Ct/f.i.Sj-.i, WP(m6_i), g A*"(a))j < e, 


m, s.l. { (V,," , Zj*, Wf (m')) £ ^*"(0)} n 
{(nGl.Zf'-i, e <"(Q)})] < e. 

E. Sketch of Converse of Theorem W.l\ 

Consider code c(n) G C with small eiTor probability Ve{c). 

i = l 

- J^^^^YfiZT+vUi.SilU'-fS'-^) (36) 

i=l 

= E I{Wi,i,W2,i;Yi,Zi) - E I(W2,i-,Ui,Si\W-i_,i) (37) 

i=l i=l 

< n- max (i(Wi,W 2 ;Y,Z) - I(W 2 -,U,S\Wi)\ (38) 


Eq. dJST l comes from taking the maximum over the set Qe- 
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